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PREFACE 


This  report  is  the  final  technical  report  (CDRL  Item  A003)  for  the 
Software  Data  Acquisition  contract.  Number  F30602-76-C-0140.  It  presents 
results  of  a project  to  collect  historical  software  development  data  from  the 
records  of  development  of  a large  Department  of  Defense  ground-based  system. 
It  includes  a general  description  of  the  subject  systems  software  character- 
istics, the  software  development  approach  and  the  software  tools  that  were 
used.  Qualitative  and  quantitative  data  gathered  from  configuration  manage- 
ment files  are  presented.  Software  reliability  model  development  and  evalua- 
tion is  expected  to  be  a primary  use  of  this  data  and  therefore,  a summary  of 
project  characteristics  useful  to  the  modeling  task  is  also  included. 

The  following  personnel  participated  in  this  project: 
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T.  James 
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EVALUATION 


The  mandate  for  producing  reliable,  maintainable  and  quality  software, 
has  been  expressed  in  various  "studies"  and  "working  groups,"  that  have  been 
generated  by  different  departments  of  DOD.  In  addition,  there  have  been 
other  meetings  held  concerning  the  same  topics,  with  participation  of  indi- 
viduals from  concerned  DOD  organizations.  As  a result,  the  requirement  for 
devising  methods  to  analyze  software  error  data  to  attain  these  goals,  has 
continually  surfaced  as  a need  that  has  to  be  dealt  with.  However,  recent 
error  data  analysis  has  been  deterred  by  the  lack  of  ample  data  from  large 
software  developments,  that  can  be  utilized  for  analysis  as  well  as  in  soft- 
ware model  testing. 

This  effort  was  undertaken  in  response  to  these  needs  and  lack  of  soft- 
ware error  data.  It  fits  into  the  goals  of  RADC  TPO  No.  5,  Software  Cost 
Reduction  (formerly  RADC  TPO  No.  11,  Software  Sciences  Technology);  specif- 
ically in  the  area  of  Software  Quality  (Software  Data).  The  report  presents 
results  of  collecting  software  error  data  from  the  records  of  a large  DOD 
ground-based  software  development  project.  The  significance  of  obtaining 
this  data,  is  that  it  will  be  used  to  support  current  software  model  develop- 
ment projects  as  well  as  be  analyzed  with  the  goal  of  developing  software 
measurements.  By  utilizing  this  data  as  stated,  it  is  expected  that  we  will 
be  better  able  to  determine  the  causes  of  software  errors  and  develop  means 
to  predict  and  possibly  prevent  them.  Additionally,  this  data  will  be  used 


IX 


along  with  other  acquired  software  error  data,  to  aid  in  establishing  a base 
line  for  ground-based  software  projects  in  quantitative  terms.  This  type  of 
information  will,  in  the  future,  lead  to  better  methods  of  developing  ground 
based  software  projects. 


JAMES  V.  CELLINI,  Jr. 
Project  Engineer 
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1.  INTRODUCTION 


This  is  the  final  report  of  a task  which  provided  a software  error  data 
base  to  be  used  in  support  of  further  research  in  software  error  analysis  and 
software  error  prediction  model  analysis.  The  effort  provided  a complete 
error  history  from  a large  Department  of  Defense  software  development  project. 
The  subject  project  was  the  development  of  software  for  a large,  ground-based, 
radar  data  processing  dominated  system.  The  error  data  base  was  extracted 
from  2165  Software  Problem  Reports  (SPRs)  written  against  109  operational 
software  modules.  The  data  base  developed  by  this  task  consists  of  three 
files,  viz: 

1)  Module  Description  File  (109  entries) 

2)  Software  Problem  Report  File  (2165  entries) 

3)  Error  Category  File  (193  entries) 

The  task  included  assigning  each  of  the  SPRs  to  one  of  the  error  types 
contained  in  the  error  category  file.  This  fault  taxonomy  is  a modification 
of  one  developed  by  TRW  as  reported  in  Reference  1.  This  report  discusses 
the  modifications  made  to  the  fault  taxonomy  and  makes  recommendations  for 
further  usage. 

The  subject  project  was  an  advanced  development  phase  project  whose  pur- 
pose was  to  demonstrate  new  concepts.  The  software  development  was  a formal 
process  with  full  documentation  required.  Engineering  change  order  (ECO) 
control  was  used  for  all  software  and  its  documentation  from  unit  release  to 
operational  (demonstration)  testing.  SofLware  Modification  Notices  (SMNs) 
were  written  to  close  out  each  opened  SPR.  This  formality  resulted  in  a very 
successful  project  and  produced  a wealth  of  documentation  which  formed  the 
basis  for  this  data  base  generation  effort. 

Because  one  of  the  problems  of  software  reliability  modeling  is  the  sim- 
plistic assumptions  made  about  the  software  development  and  testing  process, 
this  report  includes  discussions  which  are  intended  to  assist  the  model  users 
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and  developers  in  placing  the  error  data  base  in  context  of  the  software 
development  process  (Section  2),  the  type  of  operational  software  and  its 
modularity  (Section  3),  the  tools  used  (Section  4),  and  the  testing  process 
(Section  5).  The  data  base  section  (Section  6)  discusses  the  data  collected 
and  provides  additional  summary  and  statistical  information.  Recommendations 
(Section  7)  are  made  with  respect  to  the  data  collection  process,  the  fault 
taxonomy,  and  the  modeling  process. 


2.  SOFTWARE  DEVELOPMENT  PROCESS 


Figure  2-1,  the  Software  Development  Process,  provides  an  overview  of  the 
process  followed  during  the  development  of  the  software  for  the  subject  proj- 
ect. All  activity  flowed  from  the  system  requirements.  These  were  developed 
by  a System  Engineering  group  who  also  developed  the  software  requirements 
with  the  aid  of  senior  software  engineers.  Software  requirements  were  devel- 
oped and  released  for  design  in  several  functional  packages  over  a two  year 
period.  This  lengthy  "requirements  phase"  resulted  in  considerable  redesign 
which  contributed  to  the  high  percentage  (35  percent)  of  SPRs  prompted  by 
changes  in  requirements. 

Following  the  release  of  a set  of  requirements,  the  software  functional 
specification  would  be  updated  to  reflect  the  new  requirements  and  softwa 
modules  would  be  identified  and  described  functionally.  Next,  a design  speci- 
fication for  each  software  module  was  developed  and  the  "module"  or  "program 
unit"  was  then  tested  and  released  for  integration.  Figure  2-2  is  the  release 
notice  that  is  filed  when  such  a release  takes  place.  The  module  then  enters 
build  integration  testing.  This  integration  phase  was  responsible  for  the 
largest  number  (1984)  of  SPRs  of  any  of  the  test  phases.  Integration  testing 
is  the  testing  of  program  modules  with  the  system  executive  and  the  system 
data  base.  This  constitutes  a build.  Following  successful  integration  test- 
ing, the  build  was  then  released  (see  Figure  2-3)  for  acceptance  testing. 

This  took  place  at  the  hybrid  test  facility  or  at  the  demonstration  site. 
Acceptance  testing  accounted  for  a very  small  number  of  SPRs  (19).  Follow- 
ing acceptance  testing  the  build  was  released  for  operational  demonstrations. 
SPRs  were  filed  for  any  problems,  changes,  or  suspected  problems  to  a program 
unit  after  that  unit  had  been  released  for  integration  testing. 

Figure  2-4  is  the  SPR  form.  It  may  be  filled  by  anyone,  e.g.,  systems 
analyst,  programmer,  or  user  of  the  software.  The  program  unit  author  may 
issue  SPRs  against  his  own  program  unit  to  alert  others  to  deficiencies  under 
correction. 
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PROGRAM  UNIT  RELEASE  NOTICE 


I.  IDENTIFICATION 

ACRONYM  

TITLE  

MACHINE  AREA 

CONTRACT/PROJECT  

CUSTOMER  


II.  DOCUMENTATION 
REQUIREMENTS 
FUNCTIONAL  DESIGN  SPEC 
DETAILED  DESIGN  SPEC 
ACCEPTANCE  TEST  PLAN 
ACCEPTANCE  TEST  PROC 


VERSION  . 


DOCUMENT  NO. 


PHASE  DM ( ) ED(. 

RELEASE:  INITIAL  _ 

PROGRAMMER 

BUILD  


DOCUMENT  NO . 


ACCEPTANCE  TEST  RESULTS 
TEST  RESULTS  DATA 
MAINTENANCE  MANUAL 
USERS  MANUAL 
LISTING 


III.  PROGRAM  MEDIA 

ASSOCIATED  COMPOOL  

ASSOCIATED  INITIAL  CONDITIONS. 

SOURCE  TAPE  NO./FILE  NO 

CARD  DECK  (DATE)  

ED  JOVIAL  KEYWORDS  

(if  appropriate)  


TAPE  NO./FILE  NO._ 

OBJECT  TAPE  NO. /FILE  NO. 
CURRENT  LISTING  ( DATE) 


IV.  CAPABILITY 

A)  DESCRIPTION: 

B)  CHANGES  FROM  PRIOR  VERSION/MOD 

C)  GOVERNING  DOCUMENTS  (MEMOS) 

D)  SPR/SMN  CORRECTION  NO's. 


E)  STATUS  OF  UNIT  ACCEPTANCE 
TESTING  (CIRCLE  ONE) 

FULLY  PARTIALLY  * NONE  * 


F)  TESTED  WITH  ALL  REQUIRED 
HARDWARE  (CIRCLE  ONE) 

YES  NO 

DATE  INITIALS 


INITIAL 

RELEASE 

□ 

FINAL 

RELEASE 

□ 


Figure  2-2  - Program  Unit  Release  Notice 
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ATTACHKEMT  D 
BUILD  RELEASE  NOTICE 


I.  BUILD VERSION/MODV M 

SYSTEM  BUILD  TITLE 

BUILD  LEADER RELEASE:  INITIAL FINAL 

CUSTOMER  


II.  DOCUMENTATION  DOCUMENT  NO.  ER  CONTROL  NO. 

FUNCTIONAL  SPEC.  

REQUIREMENTS  

BUILD  PLAN  

TEST  PLAN  

TEST  PROCEDURE  

TEST  REQUIREMENTS  SPEC.  

TEST  RESULTS  

TEST  DATA  

USERS  MANUAL  

MAINTENANCE  MANUAL  


III.  BUILD  COMPONENTS 

A.  ASSOCIATED  COMP 00 L 

B.  PRECEDINC  BUILD(S) 

C.  THIS  BUILD  CONSISTS  OP  FOLLOWING  PROGRAM  UNITS  (Saa  Balov) 

D.  BUILD  CORE  IMAGE  TAPE  NO. 

ACRONYM  V/M  FILE  NO.  ' ACRONYM  vTfT  FILE  NO. 

1.  11. 

2.  12. 

3. 13. 


4. 


6. 


7. 


14. 

15. 

16. 
17. 


8. 

9. 
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18. 

19. 

20. 


IV.  OPERATIONS  OFFICE  CONCURRENCE 

I CERTIFY  THAT  THE  OPERATIONS  OFFICE  FILES  CONTAIN  CARD  DECKS,  MAGNETIC  TAPES,  UP-TO-DAT 
LISTINGS  FOR  EACH  OP  THE  BUILD  COMPONENTS  LISTED  IN  III  ABOVE. 


DATE OPERATIONS  OFFICE  MANAGER 

V.  INTEGRATION 

I CERTIFY  THAT  ENTRIES  IN  I,  II  AND  III  ABOVE  ARE  CORRECT.  THE  SYSTEM  BUILD  DK'ICK  I IlKI)  IN  I 
ABOVE  IS  UP-TO-DATE,  MEETS  ALL  KNOWN  SPECIFICATIONS,  AND  IS  READY  FOR  RELEASE  AS  OF  THIS  DATE. 

DATE  INTEGRATION  SECTION  MANAGER 

Figure  2-3  - Build  Release  Notice 
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APPLICATIONS  SOFTWARE  DEPARTMENT 

(SOFTWARE  PROBLEM  REPORT) 

Log  No. . 


SUBMITTED  BY:. 


Program  Unit:. 


( Signaturo  ) 


Associated  Build: _ 


Date 

Version/Mod:. 


Computer:. 


STATEMENT  OF  THE  PROBLEM:  (Type  or  Print  Plainly) 


(Describe  the  oroblc-n  both  in  programming  and  operational  terms. 
Indicate  the  manifestation  and  the  significance  of  the  problem.  ) 


PROPOSED  SOLUTION:  (If  Known  ) 


( If  Applicable  ] 


PRIORITY:  (Optional) 


Figure  2-4  - Software  Problem  Report 


CLASSIFICATION 

Design  Change 

Improvement 

C=) 

Error 

cm 

ECO  No. 

cm 

Special 

cm 

SPRs  are  generated  as  soon  as  a problem  is  identified  and  are  not  delayed 
until  a solution  is  devised  and  tested.  Their  purpose  of  to  give  technical 
and  management  personnel  early  visibility  of  problem  areas  for  earliest  solu- 
tion and  correction.  They  are  submitted  to  the  department  control  activity. 

The  department  control  activity  logs  in  the  Software  Problem  Report  and 
routes  copies  of  the  SPR  to  the  report  originator,  the  appropriate  program 
unit  author  and  his  immediate  supervisor,  integration  manager  (within  one 
working  day).  Department  Management,  designated  personnel  in  Systems  Analysis, 
and  other  specified  activities  (within  four  working  days). 

The  Software  Modification  Notice  (SMN)  shown  in  Figure  2-5  is  used  by  the 
program  author  to  log  and  correct  a specific  program  problem  which  corresponds 
to  a Software  Problem  Report.  An  SMN  may  be  issued  directly  by  a program 
author  to  correct  an  error  even  though  no  SPR  has  been  filed.  A total  of 
822  SMNs  were  filed  to  record  such  corrections.  SMNs  were  submitted  to  the 
control  activity,  with  the  corrections  properly  sequenced  to  reflect  their 
position  in  the  original  source.  SMNs  are  distributed  by  the  control  act.  vity 
in  similar  fashion  to  SPRs. 

For  each  submitted  Software  Problem  Report  the  control  activity  obtains 
a corresponding  Software  Modification  Notice  form.  For  example,  a submitted 
Software  Problem  Report  which  does  not  identify  a legitimate  program  problem 
still  must  be  closed  with  a Software  Modification  Notice  form.  The  control 
activity  insures  that  the  Modification  form  is  correctly  approved  (signed  by 
the  program  author,  Section/Group  Manager,  and  systems  integration  activity 
Manager)  when  the  change  in  implemented.  The  control  activity  maintains  the 
master  file  for  both  forms,  issues  a weekly  log  report,  and  maintains  a his- 
torical file  of  SPR/SMN  submissions  and  disposition. 
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3.  OPERATIONAL  SOFTWARE  CHARACTERISTICS 


i 0 The  subject  project  is  a real-time  control  system  for  a land-based  radar 
system.  The  operational  software  was  developed  by  Raytheon  and  executes  in 
a multiprocessor  computer  built  by  Raytheon. 

Operational  software  was  developed  in  a modular  fashion.  Nearly  all  of 
the  modules  are  written  in  J0VIAL/J3.  The  chief  exception  is  the  Executive 
program,  which,  along  with  a few  other  modules  and  subroutines,  is  written  in 
assembly  language. 

3. 1 Object  Computer  Description 

The  Raytheon  computer  consists  of  two  identical  processors  and  81,920 
words  of  24-bit  core  memory.  One  of  the  processors  is  utilized  as  a Centra] 
Processing  Unit  (CPU)  and  the  other  as  an  I/O  Control  Unit  (IOCU);  either 
processor  is  physically  capable  of  assuming  either  role  without  any  special 
reconfiguration.  Each  processor  has  its  own  set  of  internal  registers.  Both 
processors  have  common  access  to  all  primary  memory  locations. 

Each  processor  contains  two  accumulators,  two  accumulator  extension  regis- 
ters, 16  index  registers,  16  program  counter  registers,  16  pairs  of  I/O  control 
registers  and  miscellaneous  special-purpose  registers.  A repertoire  of  61 
instructions  includes  hardware  square  root  and  register-to-register  operations. 
Add  time  is  2/js.  All  arithmetic  is  fixed-point. 

Other  features  of  interest  include: 

• Unlimited  indirect  addressing 

• A "register-substitution  mode,"  which  allows  registers  other 

than  the  accumulators  to  be  specified  in  arithmetic  operations 

• A linked-list  "search  within  limits"  capability  which  automati- 
cally stacks  list  elements  successfully  meeting  the  search 
criteria 

• Special  arithmetic  instructions  for  evaluating  nested 

polynomials 

• Interprocessor  communication  capability 
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I/O  is  performed  via  16  independent ly-programmable , bidirectional  chan- 
nels. The  I/O  channels  operate  in  accordance  with  a multiplex  scheme  based  on 
channel  priority  and  channel  mode  of  operation.  A single  channel  may  be  con- 
nected to  several  individually-selectable  devices.  Data  transfers  can  be 
performed  in  either  block  mode  or  single-word  mode. 

3. 2 Data  Base  Structures 

The  subject  system  features  a common  data  base,  whose  overall  layout  is 
defined  by  means  of  a COMPOOL.  The  JOVIAL  compiler  is  COMPOOL-sensit ive,  and 
so  it  creates  at  compile  time  the  linkages  necessary  for  operational  programs 
to  gain  access  to  the  data  base. 

COMPOOL  data  is  segmented  into  blocks,  and  the  absolute  location  of  a 
particular  data  item  is  defined  in  terms  of  the  base  address  of  the  block  con- 
taining the  item  and  displacement  of  the  item  within  the  block. 

In  general,  the  compiler  generates  code  to  look  up  block  base  addresses 
in  a directory  (see  Figure  3-la).  A limited  subset  of  COMPOOL  blocks,  however, 
is  accorded  a special  status:  whenever  the  compiler  determines  that  a data 

item  resides  in  one  of  these  so-called  "special  blocks,"  it  assumes  that 
block  base  address  to  be  preset  in  a uniquely  associated  index  register  (see 
Figure  3-lb). 

Data  sets  which  are  subject  to  heaviest  use  are  assigned  to  the  special 
blocks  and  significant  reduction  in  accessing  overhead  results.  It  is  the 
responsibility  of  the  Executive  program  to  maintain  the  special  block  base 
addresses  in  the  associated  index  registers  for  use  at  run-time. 

Initialization  of  COMPOOL  data  is  accomplished  by  means  of  an  Environment 
Generation  program.  Series  of  JOVIAL  assignment  statements  are  used  to  assign 
values  to  data  items  and  thus  create  data  sets  which  can  subsequently  be 
loaded  into  memory.  All  nonvolatile  data  is  initialized  in  * h,s  fashion. 

In  addition  to  nonvolatile  data,  which  consists  of  system  parameters, 
constants  and  permanent  files,  there  are  two  classes  of  volatile  data  — 
"volatile  data  tables"  and  program  working  storage. 

Volatile  data  tables  are  used  to  contain  raw  or  processed  data  whose 
source  is  external  to  the  system  and  whose  life  span  is  relatively  short.  Radar 
input  data  is  an  example.  Application  programs  call  system  service  routines  to 
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SYMBOL  TABLE 


DATA  ITEM  ADDRESS  D • (XS) 

(b)  SPECIAL  BLOCK  SCHEME 


Figure  3-1 


Data  Accessing  Techniques 


assign  and  deassign  volatile  data  tables  of  various  types  as  necessary. 

Unused  tables  of  each  type  are  held  in  free  pools.  Table  structures  are 
defined  in  the  COMPOOL  and  allocated  to  special  blocks.  From  the  JOVIAL  com- 
piler's viewpoint,  there  is  only  a single  table  of  each  type  defined.  The 
Executive,  however,  updates  the  address  in  the  special  block  index  register 
to  link  an  application  program  to  a particular  data  table  and  thereby  makes 
that  table  the  "current"  one  of  its  type. 

Program  working  storage  is  allocated  and  deallocated  by  the  Executive  and 
is  intended  strictly  as  a local  scratch  area,  rather  than  a medium  for  passing 
data  from  program  to  program.  In  order  to  avoid  usage  conflict,  two  working 
storage  areas  are  available  — one  for  interrupt  programs  and  one  for  noninter- 
rupt  programs  (only  one  level  of  interrupt  program  is  possible) . Each  area 
consists  of  a chain  of  blocks,  with  the  first  block  provided  for  main  programs 
and  successive  blocks  provided  for  successively  nested  subroutines.  The 
JOVIAL  compiler  automatically  generates  code  requesting  working  storage  as 
part  of  the  standard  calling  sequence  for  subroutines;  the  Executive  responds 
to  these  requests  by  advancing  the  working  storage  index  pointer  to  the  next 
block  in  the  chain.  The  procedure  is  reversed  when  exiting  from  a subroutine. 
This  design  allows  reentrance. 

3 . 3 Control  St ructures and  Mechanisms 

The  subject  system  operates  under  the  control  of  a highly  centralized, 
modular  Executive  program  which  supervises  all  rea'-time  activity  on  both  the 
CPU  and  the  IOCU.  The  functional  units  comprising  the  Executive  and  described 
in  the  subsections  that  follow. 

3.3.1  Tas k Management 

This  unit  regulates  the  scheduling,  selection  and  sequencing  of  appli- 
cation program  modules.  Tasks  are  selected  for  execution  on  a priority  basis 
in  adherence  to  a limited  multiprogramming  philosophy:  The  limitation  is  that 

only  a task  of  the  maximum  priority  value  can  cause  immediate  preemption  of 
the  current  program  module;  in  the  absence  of  such  tasks,  program  modules  are 
always  allowed  to  run  to  completion.  In  order  to  assume  timely  execution  of 


all  program  modules  under  this  scheme,  application  functions  are  deliberately 
segmented  into  small,  logically  coherent  program  units.  The  Executive  uses 
a device  called  the  State  Control  Table  (discussed  below)  to  sequence  from 
one  module  to  the  next  to  form  processing  threads.  At  the  completion  of  each 
program  unit  in  the  thread,  the  Executive  checks  for  higher-priority  tasks, 
whose  presence  will  result  in  temporary  suspension  of  the  current  thread. 

New  tasks  are  scheduled  either  in  response  to  the  arrival  of  fresh 
input  data  or  in  response  to  an  explicit  request  from  a program  module.  Sched- 
uled tasks  are  placed  either  in  a "Run  Queue,"  for  execution  as  soon  as 
resources  become  available,  or  in  a "Delay  Queue,"  to  delay  execution  until 
a specified  time  interval  has  elapsed. 

3.3.2  Memory  Management 

This  unit  is  responsible  for  the  allocation  and  deallocation  of  work- 
ing storage  and  volatile  data  tables.  All  such  memory  areas  are  predefined; 
the  Executive  performs  no  dynamic  carving  of  memory. 

3.3.3  i/0  Management 

This  unit  governs  IOCU  activity,  including  coordination  and  activa- 
tion of  data  transfers  and  processing  of  external  interrupts.  It  also  reports 
the  arrival  of  new  input  data  to  the  Task  Manager. 

3.3.4  System  Auditing 

This  unit  records  information  about  program  executions,  service 
routine  usage  and  error  occurrences  in  a table  in  memory  to  assist  in  system 
performance  analysis  and  debugging. 

3.3.5  Centralized  Error  Processing 

This  unit  processes  errors  detected  by  other  software  modules  or  by 
hardware  error  traps.  Responses  vary  for  different  types  of  errors  as  dictated 
by  an  Error  Response  Table.  This  table,  moreover,  contains  two  sets  of 
responses,  one  for  the  tactical  environment  and  one  for  the  test  and  develop- 
ment environment. 
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3.3.6  System  Service  Routines 


A variety  of  system-level  subroutines  are  collected  within  the  Exec- 
utive to  eliminate  programming  redundancies  and  promote  visibility.  Functions 
provided  include  program  queuing  services,  data  management  services,  I/O 
device  handlers,  math  routines  and  miscellaneous  special-purpose  services. 

(Some  of  these  services  fall  within  other  Executive  units  as  noted  prior.) 

Sequencing  of  application  program  modules,  while  carried  out  by  the 
Executive,  is  prescribed  by  a "State  Control  T •le."  This  table  is  broken 
down  into  a number  of  sections  called  "states."  Each  state  corresponds  to  a 
single  program  module  and  consists  of  a group  of  entries  representing  all  the 
various  queuing  and  sequencing  options  for  that  module  (see  Figure  3-2). 

Two  indices  are  used  to  access  State  Control  Table  entries:  a "cur- 

rent state"  index  is  maintained  by  the  Executive;  a "condition"  index  is 
supplied  by  any  program  module  that  exits  to  the  Executive  or  calls  the  Exec- 
utive to  queue  a new  program.  These  indices  determine  a unique  table  entry, 
from  which  the  Executive  retrieves  the  identity  of  the  new  program  to  call  a 
queue,  the  new  state  associated  with  the  program,  and  the  priority  of  the 
program.  The  State  Control  Table  entry  may  alternatively  indicate  that  there 
is  no  new  program  (end-of-thread  situation),  in  which  case  the  Executive  will 
select  the  next  program  module  from  the  Run  Queue. 

The  State  Control  Table  may  be  viewed  mathematically  as  a state- 
input  device  defining  a function  of  such  tiiat,  given  a current  state  S and  an 
input  condition  C,  the  new  state  is  S'=f(S,C). 

The  State  Control  Table  enhances  modularity  by  eliminating  the  need 
for  program  modules  to  call  one  another  explicitly;  program  module  control 
interfaces  are  under  centralized  management  and  can  be  modifitl  without  impact- 
ing the  program  modules.  During  the  development  phase  of  the  subject  system, 
the  State  Control  Table  facilitated  substitution  of  dummy  programs  and  driver 
modules,  and  also  proved  to  be  a convenient  tool  for  tuning  the  system  by 
adjusting  program  priorities. 
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Figure  3-2  - State  Control  Table  Structure 


3.4  Build  Characteristics 


The  method  of  construction  of  the  subject  system  was  a synthesis  of  top- 
down  and  bottom-up  techniques.  Program  module  specifications  were  derived 
from  the  top-down,  beginning  with  system-level  requirements  and  progressing 
through  functional  and  detailed  design  specifications. 

The  highest  level  component  of  the  system,  the  Executive,  was  the  first 
program  designed  and  the  first  to  be  up  and  running.  Beyond  providing  the 
control  functions  and  services  described  above,  the  Executive,  in  conjunction 
with  the  State  Control  Table,  served  in  a broader  sense  as  a development 
medium  for  the  rest  of  the  operational  software. 

Within  the  framework  and  ground  rules  established  by  the  Executive,  inte- 
gration of  the  remainder  of  the  system  was  performed  in  a rigorously  controlled 
series  of  incremental  steps  called  "builds."  The  initial  builds  consisted  of 
groups  of  functionally  related  program  modules.  More  advanced  builds  were 
formed  by  combining  elementary  builds  and  introducing  additional  new  modules. 
The  last  build  in  the  sequence  was  the  fully  integrated  system. 

Each  build  represented  an  increment  in  hardware  capability  as  well  as 
software  capability.  The  purpose  of  a particular  build  was  not  only  to  check 
the  interrelationships  among  the  component  software  modules,  but  also  to  check 
program  interfaces  with  new  hardware  (some  of  which  was  itself  being  tested 
for  the  first  time  under  realistic  conditions). 

Program  modules  which  were  not  part  of  a given  build  were  replaced  with 
dummy  modules.  Driver  programs  performed  whatever  functions  ’-'ere  necessary 
to  keep  the  system  cycling  smoothly.  Owing  to  the  modular  nature  of  the 
system,  early  builds,  such  as  the  initial  radar  and  display  builds,  were 
functionally  independent  to  a significant  degree  and  thus  were  able  to  be 
developed  in  parallel. 
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4.  SUPPORT  SOFTWARE  CHARACTERISTICS 


A modest  array  of  software  development  tools  were  used  in  the  production 
of  the  subject  project's  operational  software: 

• Cross  Compiler 

• Compiler  Support  Software 

• Cross  Assembler 

• Digital  Simulator  of  the  Object  Computer 

• Operating  System  with  a Debugging  Package 

• Digital  System  Simulator 

• Data  Collection/Data  Reduction  Software 

Much  of  the  software  was  developed  at  a dedicated  software  development 
facility  using  a UNIVAC  1108  as  the  host  computer.  All  of  the  above  mentioned 
software,  except  for  the  operating  system,  executed  on  the  1108.  Software 
development  and  maintenance  statistics  for  these  software  development  tools 
are  not  included  in  the  software  reliability  data  base,  but  brief  descriptions 
of  each  of  these  tools  follow  to  provide  a more  complete  understanding  of  the 
software  development  process  of  the  subject  project. 

4 . 1 Cross  Compiler 

The  Higher  Order  Language  specified  for  use  in  the  subject  project  was 
J0VIAL/J3.  J0VIAL/J3  is  the  standard  programming  language  for  Air  Force 
Command  and  Control  Applications  (Reference  3).  As  a general  purpose  procedure 
oriented  language,  JOVIAL  has  been  widely  used  for  many  other  types  of  appli- 
cations. It  has  been  used  by  all  three  services.  A cross  compiler  for 
J0VIAL/J3  was  implemented  on  the  host  computer  to  produce  binary  code  for  the 
object  machine.  The  computer  implemented  the  full  J3  standard  except  for  the 
features  listed  on  the  following  page. 
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Boolean  Items 
Dual  Items 
Exchange  Operator 
Alternative  Statement 
Input/Output  Commands 


The  compiler  does  allow  embeu  led  direct  code  and  this  feature  was  used 
extensively  in  eight  of  the  subject  programs.  These  programs  have  been 
identified  as  DIRECT  (rather  than  JOVIAL  or  ASSEMBLER)  and  consist  of  at  least 
50  percent  assembly  language  embedded  in  a JOVIAL  program.  (See  Appendix  B.) 

All  system  input/output  was  centralized  in  the  executive  program,  thus 
relieving  the  JOVIAL  programmer  of  this  aspect  of  coding. 

The  average  processing  rate  of  this  compiler  is  33  source  statements 
per  second,  including  the  use  of  the  COMPOOL  (central  data  base  definition) 
and  the  generation  of  Set/Used  information. 

Appendix  E contains  statistics  about  the  static  occurrence  of  various 
elements  of  the  JOVIAL  language  taken  from  a sample  of  9 programs  from  the 
subject  project. 

4 . 2 Compiler  Support  Software 

The  JOVIAL  Compiler  Support  Software  consists  of  the  following: 

• (Communications  Pool)  COMPOOL 

• COMPOOL  Assembler 

• COMPOOL  Disassembler 

• Data  Base  Picture  Generator 

• Environment  Generator 

• Source  Library 

• Source  Reformatting  Program 

• Set /Used  Program 

Figure  4-1  depicts  the  relationships  of  these  support  programs.  The 
COMPOOL  Assembler  is  used  to  create  and  maintain  the  COMPOOL.  The  COMPOOL  is 
the  system  data  base  description  and  contains  the  global  data  item  definitions. 
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primary  memorv  mapping  information,  and  parameter  information  for  system 
subroutines.  It  is  used  by  the  JOVIAL  Compiler  and  also  used  by  environment 
generation  and  data  reduction  software.  The  COMPOOL  Disassembler  produces 
formatted  listings  and  summaries  of  the  COMPOOL  contents  to  aid  in  the  manual 
housekeeping  of  the  data  base.  The  Data  Base  Picture  Generator  provides  a two- 
dimensional  graphic  listing  of  the  data  base  and  is  useful  in  maintaining 
densely  packed  or  overlayed  data. 

Data  may  be  generated  for  initial  conditions  or  for  testing  by  the 
Environment  Generator  software  which  accepts  symbolic  test  data,  converts  it  to 
object  code  using  the  COMPOOL,  and  creates  a load  file  ready  for  use. 

The  Source  Library  contains  subroutines  for  inclusion  directly  in  a source 
module  prior  to  compilation. 

The  Source  Reformatting  Program  produces  well  formatted,  indented  listings 
and  will  optionally  resequence  the  source  file. 

The  Set/Used  Program  is  actually  an  optional  pass  of  the  JOVIAL  Compiler 
and  provides  information  on  which  data  items  are  set  (updated)  and/or  referenced 
(used)  by  the  compiled  program. 

4 . 3 Cross  Assembler 

To  provide  the  capability  for  generating  programs  at  the  instruction  level, 
a cross  assembler  was  developed.  Since  the  IOVIAL  Compiler  produced  no  code 
to  support  input-output  processing,  multiprocessing  control,  diagnostic  code 
sequences,  and  special  instructions*,  assembly  language  was  used  in  these 
instances.  The  cross  assembler  was  created  by  utilizing  the  PKOC  statement  of 
the  UNIVAC  assembler  to  develop  a macro  for  each  object  computer  instruction. 
Thus,  the  cross  assembler  was  a simple  extension  of  the  UNIVAC  Assembler  with 
a format  conversion  added  to  provide  the  proper  binary  formatted  output  for 
loading  into  the  object  machine.  The  advantage  of  this  approach  is  a rapidly 
and  inexpensively  developed,  highly  reliable  assembler.  The  disadvantage  is 
that  the  macro  processing  of  instructions  is  relatively  slow,  yielding  an 


*e.g.,  a linked  list  search/compare  instruction  was  used  for 
of  track  data. 
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assembler  that  averages  11  lines  of  source  input  processed  per  second.  This 
is  one-third  the  rate  of  the  JOVIAL  compiler;  less  if  object  instructions  are 
compared . 

4 . 4 Digital  Simulator 

Unit  testing  of  individual  program  modules  was  not  generally  done  on  the 
object  machine,  but  via  a digital  simulator  of  it,  which  executed  on  the 
UNIVAC  1108.  The  simulator  was  more  accessible  to  the  individual  programmer 
because  of  the  limited  availability  of  the  object  computers.  In  addition,  the 
fidelity  of  simulation  was  excellent  and  extensive  debugging  capabilities  were 
provided.  All  instructions  were  simulated  except  for  Input/Output  and  Multi- 
processor Control  instructions.  This  exception  did  have  an  impact,  as  the 
highest  incidence  of  SPRs  were  written  for  problems  relating  to  Input/Output . 

The  job  control  language  for  the  digital  simulator  was  syntactically 
identical  to  the  object  machine  operating  system  control  language  and  most 
of  the  commands  were  provided.  This  allowed  most  unit  tests  developed  o.  the 
simulator  to  be  executed  without  alteration  on  the  object  machine.  The  effect 
of  this  on  testing  was  not  measured  but  was  believed  to  be  highly  beneficial. 

4 . 5 Operating  System 

The  operating  system  which  supported  software  development  for  the  object 
machine  was  not  primarily  resident  on  the  object  machine,  but  instead  resided 
on  a Honeywell  DDP-124.  The  DDP-124  was  linked  via  direct  memory  access  to 
the  object  machine.  This  support  computer  provided  an  early  test  bed  capable 
of  supporting  the  development  of  a new  object  machine.  The  DDP-124  was  also 
used  as  a real  time  Input/Output  satellite  processor  for  the  object  machine. 
The  DDP-124  Operating  System  also  provided  a program  load  cat  ability  for  the 
object  machine  and  was  used  to  host  a variety  of  debugging  aids. 

The  DDP-124  included  the  following  peripheral  devices: 

• Magnetic  Tape  Drives  (2) 

• Line  Printer 

• Paper  Tape  Reader/Punch 

• Typewriter 

• Disc  Drive 
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4.6  Digital  System  Simulator 


Integration  of  software  modules  into  builds  was  accomplished  with  the 
use  of  a large  digital  system  simulator  as  the  test  bed.  The  test  facility 
included  the  object  computer  with  its  peripherals  and  operator  stations.  The 
object  computer  was  linked  via  an  interface  device  to  a UNIVAC  1108.  The  1108 
based  digital  system  simulation  software  provided  a real  time  model  of  both  the 
radar  and  the  environment  against  which  the  object  machine  was  exercised. 

Test  scenarios  were  developed  by  hand  and  processed  by  an  environment 
preprocessor.  This  data  was  then  used  by  the  real  time  simulation  to  provide 
realistic  test  conditions  for  the  object  computer.  The  vast  majority  of  SPRs 
were  generated  during  the  integration  phase  which  occurred  in  this  digital 
simulation  environment. 

4 . 7 Data  Collection/Data  Reduction 

The  data  collection  and  data  reduction  software  provided  the  capabilitv 
for  selective  recording  of  data  in  real  time  and  selective  postprocessing  of 
this  collected  data.  This  process  was  aided  by  the  use  of  the  previously 
discussed  C0MP00L  which  provided  data  structure  and  location  information  for 
the  collection  process,  and  data  format  and  content  information  for  the  post- 
process reduction. 

The  data  collector  executed  under  control  of  the  real  time  executive 
module  and  selectively  recorded  data  before  and/or  after  program  module 
execution.  The  data  was  recorded  on  magnetic  tape  for  later  reduction  on  the 
1108. 


5.  TEST  METHODS 


Testing  of  the  subject  system  was  performed  in  conformance  with  a 
meticulously  planned  and  structured  regimen.  The  overall  approach  to  testing 
closely  paralleled  the  combined  top-down/bottom-up  approach  described  in 
Subsection  3.4  for  system  integration. 

Testing  proceeded  in  three  phases:  unit  testing  of  individual  program 

modules,  including  the  Executive  program;  integration  (build)  testing;  and 
operational  testing  of  the  system  in  the  field. 

5 . 1 Unit  Testing 

The  first  stage  of  testing  was  unit  testing  of  individual  program  modules. 

In  accordance  with  the  Software  Management  Plan  for  the  subject  system  a Test 
Plan  was  conceived  for  each  program  module  as  it  was  being  developed.  The 
purpose  of  the  Test  Plan  was  to  outline  the  tests  necessary  to  demonstrate 
that  the  module  fulfilled  its  functional  requirements  and  to  verify  the 
module's  logical  integrity. 

When  the  design  of  a particular  program  module  was  completed,  a detailed 
Test  Procedure  was  produced.  Based  on  the  parent  Test  Plan,  the  Test  Procedure 
spelled  out  the  specific  techniques  to  be  used  in  the  tests,  and  included  lists 
of  input  and  output  data  as  well  as  step-by-step  instructions  for  performing 
the  tests.  The  Test  Procedure  also  described  test  driver  program  functions; 
such  functions  typically  included  interfacing  with  the  test  operator,  simulating 
interfaces  with  other  modules,  and  data  base  reinitialization  between  test  cases. 

Unit  testing  was  carried  out  on  the  Digital  Simulator  (see  Section  4) 
rather  than  the  live  computer  in  order  to  take  advantage  of  the  simulator's 
extensive  repertoire  of  debugging  tools,  including  a full  instruction  trace 
capability.  An  additional  benefit  of  this  approach  was  to  conserve  live 
machine  time,  which  became  an  increasingly  precious  commodity  as  system 
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development  progressed.  The  Simulator  not  onlv  proved  entirely  adequate  for 
unit  testing  of  application  program  modules,  but  was  also  utilized  successfully 
in  later  stages  of  testing  to  help  debug  system  problems. 

Unit  testing  of  the  Executive  program  deviated  slightly  from  the  standard 
pattern  in  that  it  was  further  subdivided  into  testing  stages  of  its  own,  and 
was  performed  on  the  live  computer  as  well  as  the  simulator.  Due  to  its  com- 
plexity, the  Executive  was  tested  at  the  individual  routine  level,  and  at  the 
fully  interactive  level,  where  it  operated  as  a skeletal  version  of  the  system. 
Because  system  I/O  is  one  of  the  Executive's  principal  functions,  and  because 
the  simulator  was  weak  in  the  I/O  area,  the  Executive  unit  tests  performed  on 
the  simulator  were  repeated  on  the  actual  computer.  This  dual  testing  approach 
also  provided  an  opportunity  to  use  the  Executive  as  a benchmark  to  evaluate 
the  accuracy  with  which  the  simulator  modeled  the  computer's  behavior. 

In  most  cases,  unit  testing  of  program  modules  was  performed  by  the  pro- 
gram authors.  After  a module  had  successfullly  passed  its  unit  tests,  it  was 
formally  released  to  an  integration  team  for  incorporation  into  a software 
build . 

5 . 2 Integration  Testing 

Integration  was  performed  in  a series  of  "builds"  as  described  in  Sub- 
section 3.4.  Each  build  was  tested  separately  in  a manner  specified  by  its 
associated  Test  Plan  and  Test  Procedure  (counterparts  to  the  program  module 
Test  Plan  and  Test  Procedure).  Because  of  the  complex  hardware  interfaces 
required  (whether  actual  or  simulated),  all  build  testing  took  place  on  a real 
machine . 

Several  facilities,  each  with  a computer  but  otherwise  featuring  different 
hardware  complements,  were  provided  to  support  integration  testing.  All  builds 
were  initially  tested  at  a software  facility  which  contained  a minimum  hardware 
conf igurat ion  (computer,  peripherals,  display  unit)  supplemented  by  a large 
scale  simulation  program  to  take  the  place  of  the  remaining  hardware  and 
simulate  the  physical  environment.  The  simulation  program  ran  in  a separate 
computer,  which  was  connected  to  the  tactical  computer  by  means  of  a special 
interface  device. 
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The  chief  purpose  of  integration  testing  at  the  software  facility  was  to 
check  out  control  and  data  interfaces  among  the  program  modules  comprising  the 
build.  A special  Executive  service  allowed  temporary  suspension  of  real  time 
processing  in  order  to  return  control  to  a build  test  driver  program  for 
varying  test  parameters  or  interacting  with  the  operator.  Test  driver  modules 
and  dummy  modules  were  also  employed  to  fill  processing  gaps  left  by  programs 
which  were  not  included  in  the  build. 

After  successful  completion  of  integration  testing  at  the  software 
facility,  a build  was  released  to  a facility  which  contained  the  actual  hard- 
ware of  central  interest  to  the  build;  other  hardware,  where  needed,  was 
simulated  by  various  means.  The  integration  tests  were  repeated  at  the  hard- 
ware facility,  this  time  to  check  out  interfaces  between  build  software  and 
pertinent  hardware  components.  Acceptance  testing  was  done  at  this  facility. 


Following  successful  integration  testing,  the  more  advanced  builds, 
including  the  full-scale  system,  were  released  as  integrated  hardware/software 
packages  for  operational  testing  in  the  field. 

Operational  testing  consisted  of  a series  of  increasingly  demanding 
missions  designed  to  exercise  the  system  and  evaluate  its  response  under  various 
loads  and  in  different  physical  environments.  Operational  missions  were  first 
rehearsed  in  conjunction  with  a Mission  Simulator,  then  performed  with  a full 
hardware  complement  under  actual  field  conditions. 
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6.  DATA  BASE 


This  section  describes  the  subject  project  data  base  development  task, 
discusses  the  data  base  contents,  and  supplies  supplementary  information  use- 
ful in  interpreting  the  data. 

6 . 1 Data  Base  Development  Task 

The  Application  Software  Department  at  the  Bedford  Laboratories  has  col- 
lected a file  of  approximately  10,000  SPR/SMNs.  The  format  and  use  of  these 
was  discussed  in  Section  2.  The  first  task  was  to  extract  each  of  the  SPR/ 
SMNs  belonging  to  the  subject  project  from  the  central  file  and  reproduce  it 
for  use  in  the  categorization  task.  Two  files  were  then  defined  to  constitute 
the  data  base  (the  third  was  added  later).  The  SPR  file  was  defined  has  1 on 
a format  used  by  TRW  for  the  Project  3 data.  Changes  were  required  because 
additional  data  was  being  collected  and  some  data  items  were  deleted.  The 
second  file  defined  was  the  software  module  file  which  was  to  contain  the 
characteristics  of  the  software  modules  against  which  the  SPRs  were  written. 
See  Appendix  A for  a detailed  format  of  each  of  these  files.  Each  SPR/SMN 
was  then  reviewed  by  a programmer  who  had  worked  on  the  subject  projects 
integration  task,  and  an  error  category  was  assigned  using  the  TRW  fault 
taxonomy  as  presented  in  Table  4-1  of  Reference  1.  Several  programmers  worked 
at  this  task  which  required  about  seven  man/months  to  complete.  Over  2400 
SPR/SMNs  were  reviewed.  Other  historical  documentation,  some  on  microfilm 
files,  were  then  reviewed  and  data  on  module  Characteristics  were  extracted. 

At  this  point  the  data  was  keypunched  and  placed  on  a computer  for  editing. 

A program  was  written  to  match  the  module  description  file  against  the  SPR/ 

S'".'  file  to  correlate  program  names.  This  nrogram  also  presented  formatted 
itnut  and  did  some  editing  of  the  data  (see  Appendices  B and  Cl.  At  this 
-joint  a third  file  was  developed  which  contained  the  error  categories. 
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file  was  used  to  verify  that  the  error  category  codes  on  the  SPR/SMN 
valid  (see  Appendix  D) . Later  code  was  added  to  accumulate  the 
SPRs  written  against  each  program  module  and  against  each  error 
Statistical  routines  were  then  added  to  produce  summary  statistics, 
fourth  file  was  developed  and  a code  was  added  to  translate  the  sub- 
ct's  program  module  names  into  innocuous  names  to  preserve  project 


6 . 2 Data  Base  Contents 

The  resulting  data  base  as  delivered  to  RADC  consisted  of  the  three  files 
whose  formats  appear  in  Appendix  A.  Each  will  be  briefly  discussed  in  this 
section.  Those  data  items  requiring  interpretation  are  specifically  discussed. 

6.2.1  Software  Module  Descriptions  (Refer  to  Appendices  A and  B) 

This  file  consists  of  109  entires,  each  containing  the  characteristics 
of  an  individual  program  module.  Ther  version  identification  shown  is  that  of 
the  last  released  version/modification  of  that  particular  program.  The  version 
number  represents  a major  functional  release  of  the  program.  Thus  version  2 
indicates  that  three  major  functional  releases  had  been  made.  The  modification 
letter  represents  the  number  of  modification  releases  (minor  functional  changes 
or  error  corrections)  within  the  version.  E represents  the  fourth  modification 
release.  PROG027  AO  would  be  the  initial  release  of  PROG027.  PROG036  4J 
indicates  that  the  program  has  had  five  major  functional  releases  and  the 
current  version  has  had  nine  modification  releases.  This  data  is  generally 
inadequate  to  allow  determination  of  the  total  number  of  releases  since  each 
version  may  have  from  no  modification  releases  to  many. 

The  next  field  indicates  the  generic  function  of  the  nodule  and  is 
somewhat  subjective  although  few  programs  were  difficult  to  assign  to  a generic 
function.  The  complexity  characteristic  was  also  assigned  in  a subjective 
fashion,  although  again  no  difficulty  was  encountered  in  assigning  complex  or 
simple  to  a module.  Mode  of  construction  was  limited  to  modular  or  unstructured, 
as  top-down  or  structured  development  was  not  used.  Appendix  B contains  a 
complete  listing  of  the  module  description  file. 
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Table  6-2  contains  the  distribution  of  SPRs  by  module 
distribution  of  module  types. 
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TABLE  6-2 

DISTRIBUTION  OF  SPRs  BY  MODULE  TYPE 


Module  Type 

Percent  of  Total 

Percent  SPRs 

Logical 

20.  2 

9.  6 

Control 

8.  3 

9.  5 

Mathematical 

19.  3 

18.  7 

I/O 

5.  5 

5.  0 

DATA  BASE 

8.  3 

17.  5 

Microcode 

0.  9 

1.  3 

1 

i COMPOOL 

0.9 

2.  3 

Data  Manipulation 

11.  0 

18.  4 

j Test  Driver 

5.  5 

10.  3 

This  table  reveals  that  the  DATA  BASE  modules  should  have  been  given 
more  attention.  The  DATA  BASE  modules  for  the  subject  project  are  not  data 
base  definitions  (that  is  the  COMPOOL)  but  are  initial  conditions  for  a build. 
Perhaps  better  tools  could  have  helped  here.  One  problem  with  this  table  is 
that  the  size  of  the  modules  is  not  taken  into  consideration. 

Table  6-3  shows  the  number  of  SPRs  normalized  to  1000  lines  of 
source  code. 


TABLE  6-3 

SPRs  NORMALIZED  TO  1000  LTNES  OF  SOURCE 


SPRs/1000  Lines  of  Source 


Percent  of  Total  Size 


r 
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The  five  module  types  represent  the  operational  executable  modules 
and  were  ratioed  to  100  percent.  The  relatively  low  figure  for  the  control 
module  can  be  attributed  to  the  fact  that  significant  portions  of  the  real 
time  executive  program  were  derived  from  a previous  project. 

6.2.2  Software  Problem  Report  File  (Refer  to  Appendices  A and  C) 

The  SPR  file  consists  of  2165  entries  each  containing  data  on  a 
single  SPR/SMN  pair  or  SMN  only,  if  no  SPR  was  filed.  Note  that  the  SPR 
numbers  are  not  a dense  set  since  they  are  not  project  specific.  The  termina- 
tion code  is  "SOFTWARE"  if  an  unexpected  test  termination  attributed  to  a soft- 
ware problem  was  specifically  mentioned  on  the  SPR:  similarly  "hardware"  for 
hardware  problems  which  caused  an  unexpected  test  termination  which  was  thought 
to  be  software  (thus  an  SPR  was  filled  out)  but  later  attributed  to  hardware. 

Of  the  2165  SPRs,  47  resulted  in  specifically  identified  unexpected  software 
terminations  and  seven  resulted  in  specifically  identified  unexpected  hardware 
terminations  originally  though  to  be  software  problems.  The  seriousness  if 
the  error  was  determined  to  be  CRITICAL  if  the  discoverer  indicated  that  it 
was  impeding  project  development,  LOW  if  it  was  not  really  necessary  for  a 
correction  to  be  made  for  the  current  development  to  proceed,  IMPROVEMENT  if 
it  was  a suggestion  for  improvement  but  not  necessary  for  satisfactory  opera- 
tion, and  MEDIUM  otherwise.  Table  6-4  lists  the  occurrence  of  each  of  these 
levels  of  seriousness. 


TABLE  6-4 

SERIOUSNESS  OF  SPRs 


Seriousness  Type 

Number 

Percent  of  Total 

Critical 

134 

6.  2 

Medium 

1642 

75.  8 

Low 

105 

4.  9 

Improvement 

285 

13.  1 

The  test  periods  of  concern  to  this  data  base  are  the  Integration, 
Acceptance,  and  Operational  periods.  Integration  occurs  following  unit 
development  and  formal  release,  and  occurred  at  a software  development  facility. 
Acceptance  tests  were  then  run  at  a hybrid  test  facility.  SPRs  which  speci- 
fically mentioned  acceptance  testing  or  were  known  to  be  found  during  accep- 
tance testing  by  integration  programmers  were  identified  as  Acceptance  SPRs. 

All  SPRs  filed  from  the  operational  site  were  identified  as  Operational  SPRs. 
Table  6-5  lists  the  occurence  of  SPRs  during  each  of  these  periods. 

TABLE  6-5 

OCCURRENCE  OF  SPRs 


Test  Period 

Number 

Percent  of  Total 

Integ  ration 

1984 

91.6 

Acceptance 

19 

0.  9 

Ope  rational 

162 

7.  5 

The  error  category  code  is  the  code  indicating  the  error  category  as 
listed  in  file  3 (see  Appendix  I)). 

The  SMN  number  should  in  all  cases  be  the  same  as  the  SPR  number; 
except  that  some  clerical  errors  were  made  during  the  original  assignment  of 
numbers.  Cases  of  this  are  indicated  by  an  * to  the  right  of  the  SPR  number. 
As  mentioned  in  Section  2,  some  SMNs  were  tiled  without  a corresponding  SPR. 
These  were  usually  the  result  of  a programmer  discovering  th>_  error,  correct- 
ing it,  and  then  issuing  an  SMN  to  release  the  correction.  A total  of  822 
SMNs  (38  percent)  were  filed  without  SPRs. 

The  Correction  Type  indicates  the  tvpe  cf  change  or  update  made  as 
a result  of  the  SMN.  Unfortunately  this  data  was  not  generally  captured  and 
is  insufficient  for  statistical  use. 

The  Days  Open  data  was  extracted  from  the  Raytheon  Manufacturing 
Davs  calendar  and  represents  the  number  of  working  days  between  the  date  open 
and  date  closed.  SMNs  filed  without  SPRs  were  set  to  1 day  opened. 
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The  2165  SPR/SMN9  were  opened  for  a total  of  17,015  days,  or  an 
average  of  7.9  days.  This  Is  distorted  somewhat  by  the  relatively  high  per- 
centage of  SMN-only  reports.  Removing  the  SMN-only  reports  yields  1343  SPR/ 
SMNs  opened  for  a total  of  16,193  days  or  an  average  of  12.1  day9. 

Because  of  the  file  length  only  a small  portion  is  included  in 
Appendix  C.  RADC  does,  however,  have  the  entire  file. 

File  6-1  shows  the  distribution  of  the  SPR/SMNs  by  month  opened 
during  the  38  months  of  integration  through  operational  testing. 

The  curve  peaks  at  133  SPRs  opened  during  month  5 of  the  second  year, 
and  drops  to  a low  of  three  opened  during  month  10  of  the  third  year. 


6.2.3  Error  Category  File  (Refer  to  Appendices  A and  D) 

The  error  category  file  consists  of  193  entires,  one  per  error 
category.  The  error  categories  were  based  on  the  184  as  defined  by  TRW  in 
Reference  1.  Added  categories  are  flagged  with  an  asterisk  to  the  right  in 
Appendix  D.  Additions  were  made  to  categorize  the  following  errors: 

a)  Scaling 

b)  New  of  enhanced  function  - display 

c)  Modifications  for  special  test  purposes 

d)  Unidentified  hardware  error 

e)  Nonrecurring  problems 

f)  No  error 

g)  Insufficient  information  for  error  analysis 

h)  Missing  cards  (source  lines)  in  a compiled  program 

i)  Inadequate/Inefficient  requirements 

j)  Enhancement  requirements 


Table  6-6  contains  the  summary  of  SPRs  by  category  group.  Refer  to 
Appendix  D for  the  meaning  of  the  category  group  code. 

The  most  frequent  errors  by  category  group  were  the  User  Requested 
Changes  (35.3  percent),  with  Data  Handling  Errors  (18.9  percent)  and  Logic 
Errors  (17.6  percent)  making  up  the  largest  percentage  of  the  remainder.  The 
high  incidence  of  user  requested  changes  is  most  likely  a characteristic  of 
the  evolutionary  development  approach. 


TABLE  6-6 

SPRs  BY  CATEGORY  GROUP 


Category 

Group 

No. 

SPRs 

Pe  rcent 

AA 

Computational 

115 

5.  3 

BB 

Logic 

382 

17.  6 

CC 

I/O 

21 

1.  0 

DD 

Data  Handling 

409 

18.  9 

EE 

Operating  System/Support  Software 

4 

0.  2 

FF 

Configuration 

18 

0.  8 

GG 

Routine/Routine  Interface 

16 

0.  7 

HH 

Routine/System  Interface 

17 

0.  8 

JJ 

User  Inte rface 

10 

0.  5 

KK 

Data  Base  Interface 

32 

1.  5 

LL 

User  Requested  Changes 

764 

3 r.  3 

MM 

Preset  Data  Base 

162 

7.  5 

NN 

COMPOOL  Rejection 

45 

2.1 

PP 

Recurrent 

39 

1.  8 

QQ 

Comment  s 

1 5 

0.  7 

RR 

Requirements  Compliance 

10 

0.  5 

SS 

Unidentified 

77 

3.  6 

TT 

Operator 

15 

0.  7 

UU 

Que  stions 

3 

0.  1 

V V 

Requirements  Specification 

1 1 

0.  5 

f> — 9 
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This  subsection  contains  supplementary  information  of  possible  use  to 
modelers.  It  presents  an  analysis  of  build  information,  acceptance  test  data, 
and  operational  data. 


6.3.1  Build  Analysis 

As  mentioned  previously,  there  were  several  builds  implemented  during 
the  life  of  the  project.  As  a final  deliverable  item,  there  were  two  builds 
delivered.  These  builds  consisted  of  an  Initialization  Build  (Build  G)  and  an 
Operational  Build  (Build  F) . The  Initialization  Build  performed  hardware 
diagnostics,  hardware  and  software  confidence  test,  and  initialized  both  hard- 
ware and  software  data  bases.  The  Operational  Build  was  comprised  of  55  pro- 
gram modules  which  were  implemented  and  tested  in  Builds  A through  E and  then 
put  together  as  a system.  Appendix  F contains  the  list  of  program  modules 
for  those  two  builds  for  possible  use  in  further  analysis. 

During  the  life  of  the  project,  records  were  kept  to  be  used  for 
estimating  new  projects  in  the  future.  The  types  of  data  collected  were: 

• Record  of  all  software  problems  by  number  and  date 

• Amount  of  computer  time  using  wall  clock  time 

• Manpower  allocated  to  each  build  within  the  project 

The  following  subsections  discuss  the  software  problems  associated 
with  each  of  the  two  delivered  builds. 


6 . 3 . 1  . 1 Bu  i Id  "F"  DJjscuss  i on 

6 . 3 . 1 . 1  . 1 Background 

Integration  testing  of  Build  F was  performed  over  a 35 
month  period.  Within  this  time  rame , there  were  a total  of  41  releases  of 
the  build  reflecting  error  corrections,  design  changes  and  improvements. 
Months  1 through  7 were  devoted  to  testing  the  build  using  t he  Digital  Svstem 
Simulator.  During  the  next  five  months  the  build  was  tested  at  a test  site 
with  hardware  and  also  in  parallel  on  the  Digital  Svstem  Simulator. 
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It  is  appropriate  here,  to  mention  that  the  software  was 
being  tested  on  hardware  that  was  not  completely  checked  out,  thus  adding  to 
the  amount  of  time  necessary  to  resolve  problems.  Hardware  diagnostics  were 
not  sophisticated  enough  to  diagnose  all  problems  and  many  were  found  during 
operational  software  testing. 

Testing  for  the  remaining  20  months  was  accomplished  by 
first  testing  a particular  release  of  the  build  on  the  Digital  System  Simulator 
and  then  shipping  to  a field  site  for  operational  testing  on  the  hardware  in  a 
live  environment. 

During  the  entire  integration  period,  a total  of  136  man- 
months  of  effort  was  expended.  There  is  no  record  for  computer  time  used 
while  testing  with  the  hardware.  The  computer  time  (wall  clock  time)  utilized 
for  testing  with  the  Digital  System  Simulator  amounted  to  1890  hrs  and  47  min. 
See  Table  6-8  for  the  monthly  usage  of  computer  time  for  the  builds. 

6 . 3 . 1 . 1 . 2 Discussion 

In  a 35  month  period,  there  were  1198  problems  reported, 
investigated,  and  resolved.  Figure  6-2  depicts  the  number  of  problems 
reported  each  month.  After  investigating  the  file  of  problem  reports,  it  was 
discovered  that  the  peaks  and  valleys  shown  in  Figure  6-2  tracked  each  major 
release  of  the  build.  The  peaks  represent  the  time  of  build  release  when 
several  problems  had  been  resolved.  The  valleys  represent  the  end  of  testing 
particular  functions  and  preparing  to  work  on  the  next  release,  which  is  based 
on  the  results  of  the  tests  and  addition  of  new  functions  of  complicated  test 
aimed  at  final  checkout  of  the  system. 

Another  factor  which  attibuted  to  the  rise  and  fall  in 
numbers  of  problems  was  the  parallel  effort  of  hardware  integration  and  hard- 
ware downtime.  When  hardware  is  malfunctioning  or  down,  the  software  problems 
are  not  readily  found. 

Months  12  through  15  reflect  the  period  which  had  the 
largest  number  of  problems  reported.  While  reviewing  the  problem  reports,  it 
became  visible  that  the  build  during  this  time  period  was  being  tested  for 
the  first  time  at  the  field  site  in  preparation  for  the  first  mission.  During 
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the  testing,  it  became  evident  that  some  of  the  interfaces  with  site  hardware, 
which  could  not  be  tested  with  simulation  tools,  and  the  environmental  data, 
were  different  than  had  been  anticipated.  New  software  logic  had  to  be  added. 
Software  was  also  modified  to  adapt  to  environmental  interference  (ground  or 
weather  clutter)  which  was  overloading  the  system. 

After  the  15th  month  of  integration  testing  the  number  of 
software  problems  decreased,  which  also  resulted  in  a decrease  of  manpower 
levels.  In  essence,  the  remaining  months  were  devoted  to  fine  tuning  the 
system.  Software  errors  were  found  in  areas  that  had  not  been  completely 
tested  using  simulation.  However,  most  of  the  problems  were  user  requested 
changes,  product  improvements,  and  modifications  to  initial  conditions  due 
to  environmental  conditions. 

Table  6-7  lists  the  number  of  total  problems  and  the  per- 
centage of  total  problems  reported  for  each  problem  category.  It  is  readily 
observed  that  the  majority  of  problems,  in  fact  38  percent,  were  due  to  design 
changes  and  improvements.  Logic  errors  and  data  handling  errors  were  18  and 
16  percent  respectively.  These  three  categories  of  problems  constituted  the 
major  system  problems. 

It  was  rather  difficult  to  collect  data  with  respect  to  an 
individual  build  release.  For  example.  Build  F had  41  releases  and  the  pro- 
blem reports  did  not  usually  connect  a problem  to  a build  release.  To  generate 
this  report,  a great  deal  of  time  was  devoted  to  correlating  the  problems  and 
build  releases  using  supervisor  status  reports  and  bracketing  build  release 
dates  with  problem  report  dates. 


6. 3. 1.2  Build  "G"  Discussion 


6. 3. 1.2.1  Background 

Build  G had  a 37  month  span  of  integration  testing.  The 
Build  was  comprised  of  hardware  diagnostics,  hardware  confidence  tests,  and 
hardware/software  initialization  programs.  The  diagnostics  verified  the 
operability  of  the  computer  while  the  confidence  tests  verified  each  subsys- 
tem within  a radar  system  such  as,  receiver,  transmitter,  signal  processor, 
etc . 

In  developing  the  programs,  the  majority  of  them  could  be 
tested  individually  on  an  off  line  computer,  except  for  the  actual  I/O  inter- 
faces. The  hardware  interfaces  had  to  be  tested  on  the  actual  hardware  as  it 
became  available.  For  Build  G,  the  hardware  and  software  development  was 
being  performed  in  parallel.  A simulator  was  not  available  to  test  the  I/O 
interfaces . 

It  should  be  pointed  out  that  the  programs  in  this  Build 
at  the  start  of  the  system  were  designed  as  independent  programs.  It  was  not 
until  some  time  into  system  generation  that  a decision  was  made  to  automate 
the  programs  to  operate  sequentially  without  operator  intervention  as  a Build. 
Therefore,  testing  of  a majority  of  the  programs  had  been  completed  indepen- 
dently. The  Build  testing  basically  consists  of  hardware  integration  testing. 

Table  6-8  shows  the  monthly  use  of  computer  time  (wall 
clock  time)  used  to  integrate  the  software  before  testing  with  actual  hardware. 
Over  the  three  year  period,  a total  of  720  hours  and  18  minutes  were  utilized. 
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TABLE  6-8 

COMPUTER  TIME  FOR  SOFTWARE  INTEGRATION 
IN  WALL  CLOCK  HOURS 


Month  i Build  F I Build  G I Usage 


Month  Build  F I Build  G I Usage 


12:35 

19:55 

52:30 

47:11 

95:06 

55:45 

59:15 

43:35 

44:45 

42:20 

75:00 

62:50 

73:35 


57:45 

46:23 

34:10 

28:55 

12:42 

27:04 

54:12 

51:50 

50:24 


24:40 
21 :30 
27:15 


22:10 

15:35 


31:55 


127:45 

122:10 

122:52 

109:53 

97:56 

82:54 

110:23 

160:53 

150:12 


68:08  238:16 


134:20 

177:25 

121:30 

141:19 

140:20 

124:05 

94:05 

178:37 


42:45 
40:40 
96:45 
88:35 
56:45 
79:15 
73:30 
65:20 
67:50 
1 16:55 
88:05 
78:05 
37:55 
41:05 
54:30 
63:00 
39:30 


23:40 

18:15 

16:10 

16:15 

13:20 

35:40 


134:55 
157:55 
1 17:20 
121:30 
99:20 
67:55 
70:20 
1 16:55 
89:05 
78:05 
41:10 
41:05 
54:30 
63:00 
43:50 
39:30 


Note:  Months  without  computer  time  indicate  testing  performed  at 

acceptance  test  site  or  operational  site. 


6. 3. 1.2. 2 Discussion 


There  were  173  problems  and  59  man  months  of  effort 
reported  over  a 37  month  period,  which  appears  to  be  low,  compared  to  Build  F. 
However,  the  low  number  of  problem  reports  is  attributed,  on  the  most  part, 
to  only  hardware  integration  versus  the  combination  of  software  and  hardware 
integration.  The  logic  and  data  handling  errors  were  found  only  in  a few 
programs  which  had  not  been  completely  tested  on  the  hardware  prior  to  being 
put  into  the  Build. 

The  peak  months  of  problems  reported  in  Build  F occurred 
in  the  field  when  intensive  testing  and  fine  tuning  of  the  system  was  being 
performed.  In  some  instances,  data  formats  and  interface  bit  configurations 
were  changed  to  make  the  system  more  efficient.  There  were  also  changes  made 
to  software  to  bypass  hardware  fixes  which  were  more  costly. 

Figure  6-3  shows  the  errors  that  were  reported  each  month 
and  the  problem  categories  they  represented.  The  Build  was  so  dependent  on 
hardware  scheduling  that  it  is  impossible  to  generate  curves  representing 
software  reliability.  The  software  was  tested  in  spurts  over  the  37  month 
period.  The  other  variable  in  the  software  testing  was  that  all  hardware  was 
not  available  for  testing  until  late  in  the  25th  month  of  the  Build. 

While  analyzing  the  types  of  problem  reports,  there  was  a 
definite  resemblance  to  all  other  builds  with  respect  to  percentage  of  prob- 
lems by  problem  category.  Table  6-9  reflects  the  types  of  problems  and  their 
percentage  of  the  total  number  or  problems. 

Approximately  50%  of  the  problems  were  devoted  to  user 
requested  changes  or  product  improvements.  The  data  handling  errors  reflected 
22%  of  the  problems  and  logic  errors  14%.  All  remaining  problems  only  accu- 
mulated to  14%  of  the  total  problems. 
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Figure  6-3  - Build  "G"  Problem  Reports  by  Month  and  Error  Category 


TABLE  6-9 


BUILD  "G"  PROBLEM  CATEGORY  DATA 


Problem 

Number  of 

Percentage  of 

Category 

Problems 

Total  Problems 

AA 

0 

BB 

25 

14.45 

CC 

1 

0.  58 

DD 

38 

21. 96 

EE 

0 

FF 

0 

GG 

0 

HH 

9 

5.  20 

I I 

0 

JJ 

2 

1.  15 

KK 

3 

1.73 

LL 

86 

49.  71 

MM 

0 

NN 

0 

PP 

3 

1.73 

QQ 

2 

1.  15 

RR 

0 

SS 

2 

1.  15 

TT 

2 

1.  15 

UU 

0 

VV 

0 

Total 

173 

u 
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6.3.2  Acceptance  Test  Data 


Acceptance  test  data  is  sparse  and  unreliable.  Most  often  the 
authors  of  SPRs  did  not  indicate  on  the  SPR  that  the  problem  occurred  during 
an  acceptance  test.  Only  19  SPRs  were  so  marked.  This  made  it  impossible  to 
gather  significant  information  about  the  impact  of  software  problems  on  the 
acceptance  test  process  including  the  impact  on  other  testing.  There  were  a 
total  of  19  Acceptance  Test  SPRs  or  0.9%  of  the  total.  Of  the  19,  17  were 
critical,  one  was  an  Improvement,  and  one  was  Low  Seriousness.  The  17 
critical  SPRs  were  corrected  in  an  average  of  4.3  days,  with  a standard  devia- 
tion of  4.3  days.  The  distribution  of  errors  by  category  group  is  shown  in 
Table  6-10. 


TABLE  6-10 

ACCEPTANCE  TEST  ERRORS  BY  CATEGORY 


Category  Group 

Number  of  SPR's 

AA 

Computational 

2 

BB 

Logic 

3 

CC 

I/O 

1 

KK 

Data  Base  Interface 

1 

LL 

User  Requested  Changes 

1 1 

SS 

Unidentified 

1 
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6.3.3  Operational  Data 

Operational  demonstrations  took  place  at  a remote  site.  Again  data 
is  sparse  with  respect  to  the  impact  of  software  errors  on  the  entire  test 
effort.  Of  the  162  operational  SPRs,  31  were  designated  as  critical.  The  31 
critical  SPRs  were  corrected  in  an  average  of  11.6  days,  with  a standard  devia- 
tion of  11.3  days.  The  distribution  of  errors  by  category  group  is  shown  in 
Table  6-11. 


TABLE  6-11 

OPERATIONAL  ERRORS  BY  CATEGORY 


Category  Group 
AA  Computational 

BB  Logic 

CC  I/O 

DD  Data  Handling 

GG  Interface  - Routine/Routine 

HH  Interface  - Routine/System 

JJ  Interface  - User 

KK  Interface  - Data  Base 

LL  User  Requested 

MM  Preset  Data  Base 

PP  Recurrent 

RR  Requirements  Compliance 

SS  Unidentified 

TT  Operator 

UU  Questions 

VV  Requirements  Specification 


Number  of  SPR's 
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Again  the  high  level  of  user  requested  changes  reflects  the  evolution- 
ary nature  of  the  development. 

Table  6-12  indicates  the  load  placed  on  the  software  in  the  opera- 
tional environment.  This  may  be  useful  in  the  analysis  of  operational  errors. 


TABLE  6-12 

EXECUTION  LOADING  BY  MODULE  TYPE 


Module  Type 

Light  Load 

Heavy  Load 

Control 

10% 

10% 

Mathematical 

0 

44% 

Logic 

1 1% 

16% 

Data  Manipulation 

13% 

26% 

I/O 

3% 

3% 

37 % Loaded 

99%  Loai 

7.  RECOMMENDATIONS 


As  mentioned  in  the  introduction  the  intended  use  of  this  data  base  is 
to  support  the  development  of  software  reliability  models.  During  the  pro- 
cess of  building  the  data  base,  the  primary  purpose  of  this  project,  some 
thought  was  given  to  the  significance  of  the  data  and  the  uses  to  which  data 
of  this  type  might  be  put.  This  section  identifies  some  of  the  characteris- 
tics of  the  subject  project  and  data  which  may  influence  the  accuracy  of  the 
models.  Recommendations  are  also  made  with  respect  to  the  collection  of 
such  data  in  future  projects  and  the  potential  uses  of  the  data  while  it  is 
still  "fresh." 

7 . 1 Subject  Project  Characteristics  That  May  Affect  Modeling 

Several  characteristics  of  the  subject  project  may  be  of  some  interest 
to  those  constructing  software  models.  While  quantitative  data  was  not 
gathered  for  this  project,  these  characteristics  might  serve  to  assist  in  the 
selection  of  an  applicable  model  as  well  as  indicating  possible  areas  for 
future  extension  of  models.  For  the  subject  project  these  characteristics 
were : 


1)  evolutionary  development  of  software  requirements 

2)  evolutionarv  development  of  the  system 

3)  parallel  hardware  development 

4)  multiple  system  configurations 

5)  build  process 

6)  uneven  application  of  resources 

7)  previously  existing  software 

8)  lack  of  development  phase  data 


As  mentioned  earlier  in  this  report,  the  software  requirements  for  the 
subject  project  were  issued  in  several  releases  over  a two  year  period.  Due 
to  schedule  pressure,  informal  or  preliminary  releases  were  also  made.  This 
characteristic  probably  contributed  heavily  to  the  large  percentage  of  "User 
Requested"  changes  to  the  software.  Many  large  DOD  system  developments  have 
this  characteristic.  It  is  really  related  to  the  evolutionary  approach  to 
system  development  which  seeks  to  minimize  risk  by  testing  concepts  and 
evolving  the  system  in  a step-by-step  orderly  fashion.  This  approach  is  com- 
mon when  a system  is  being  developed  which  does  not  use  off-the-shelf  compo- 
nents and  proven  technology. 

Another  characteristic  of  this  project  was  parallel  hardware  development. 
Early  users  of  the  new  hardware  suffered  from  the  "serial-number  1"  syndrome 
and  the  high  incidence  of  hardware  failures  had  a pronounced  effect  on  the 
software  development.  However,  since  most  of  the  early  failures  were  imme- 
diately recognized  as  being  hardware  problems,  no  software  problem  reports 
were  filed.  The  data  was  not  captured. 

Software  developed  for  the  subject  project  was  executed  on  three  similar 
computer  configurations,  each  "slightly"  different  in  its  usage  of  input/ 
output  channels  and  its  suite  of  peripherals.  These  "slight"  differences 
contributed  to  the  high  incidence  of  Input/Output  errors.  Software  checked 
out  at  the  integration  facility  would  require  minor  modifications  in  input/ 
output  areas  when  executed  at  the  acceptance  test  facility  and  later  at  the 
operational  site.  Each  of  these  modifications  was  recorded  via  a SMN  to  main- 
tain configuration  control,  and  so  entered  the  error  data  base.  This  type  of 
"error"  should  be  filtered  out  before  using  the  data  in  a reliability  model 
as  these  modifications  are  really  adaptations. 

Another  possible  problem  for  the  modeling  effort  is  the  build  process. 

In  such  a process,  each  successive  build  jeopardizes  the  reliability  function 
( R ( t ) ) of  the  previous  build.  Therefore,  R(t)  should  increase  as  build  testing 
progresses.  Then,  at  the  next  build,  it  would  probably  decrease.  The  new 
functions  that  are  added  to  each  build  differ  in  size  and  complexity.  As  one 
would  expect,  the  simple  functions  were  integrated  before  the  more  complex 


7-2 


functions.  Therefore,  succeeding  builds  became  more  difficult  to  test 
because  of  the  larger  number  of  interconnections  and  interactions  between 
the  various  modules.  Therefore,  the  total  errors  (E^_ ) increase  with  each 
succeeding  build. 

A careful  look  at  Figure  6-1  reveals  several  sharp  dips  in  the  number  of 
SPRs  opened.  Several  of  these  occur  at  the  end  of  the  calendar  year,  the 
end  of  the  fiscal  year,  and  at  the  time  of  summer  vacation.  Most  likely,  the 
intense  activity  just  preceding  the  dip  occurred  at  a build  release  or  a major 
system  milestone  which  are  likely  to  fall  just  prior  to  these  above-mentioned 
times  and  are  followed  by  a lull  in  activity.  These  indicate  uneven  applica- 
tion of  resources,  primarily  manpower,  and  supplementary  data  on  applied 
manpower  is  needed  to  normalize  the  data  and  accurately  relate  error  dis- 
covery to  applied  effort. 

Another  area  which  affects  software  reliability  is  the  extent  to  which 
previously  developed  software  is  used.  Previously  developed  software  may 
occur  as  library  routines,  entire  programs,  or  as  published  algorithms.  It 
is  known  that  a small  percentage  of  the  software  (probably  <10%)  of  the 
subject  project  was  developed  previously,  but  the  actual  data  is  lost  in  the 
past . 

Software  error  data  from  the  development  phase  is  not  available.  Many 
of  the  error  categories  (e.g.,  compiler  errors,  job  control  errors,  etc.) 
would  show  up  predominantly  in  this  early  phase.  It  is  a reasonable  suspicion 
that  a program  with  poor  reliability  during  the  development  phase  is  likely 
to  have  poor  reliability  in  later  phases,  but  it  would  be  helpful  to  have  hard 
facts  in  this  area.  On  the  other  hand  a program  may  have  high  reliability 
during  the  development  phase  and  poor  reliability  during  integration.  This 
would  indicate  problems  in  development  testing,  or  interface  design. 

7 . 2 Data  Collection 

Reference  1 emphasizes  the  need  to  provide  accurate  error  categorizing 
at  the  time  the  error  is  identified.  To  do  this  at  a later  date  requires  some 
degree  of  interpretation  from  historical  documentation  which  can  introduce 
further  error  and  distort  the  reliability  information.  We  recommend  that 


the  programmer  who  creates  the  fix  for  the  problem  also  does  the  error 
category  assignment.  The  assigned  category  should  be  independently  verified, 
possibly  by  a software  quality  assurance  engineer.  Since  the  error  category 
assignment  does  involve  an  element  of  interpretation,  this  concurrence  would 
enhance  the  reliability  of  the  assignment. 

One  problem  with  the  fault  taxonomy  used  for  this  data  base  development 
was  the  large  number  of  categories,  some  of  which  were  overly  specific  (e.g., 
time  conversion  error).  This  overspecifying  of  error  categories  led  to  incom- 
pleteness and  it  seemed  to  us  that  a level  of  generality  was  needed  (e.g., 
conversion  error).  The  major  complaint  by  the  category  assigners  was  that 
the  number  of  categories  was  too  large  and  the  amount  of  subjectivity  involved 
in  assignment  led  to  an  uncomfortable  feeling  that  some  assignments  were 
ambiguous.  Subsequent  to  our  categorization  of  errors,  the  final  report  was 
issued  (reference  2)  and  the  number  of  categories  were  reduced  to  79,  less 
than  half  the  original  list.  (See  Table  3-2,  of  reference  2).  We  believe 
that  this  taxonomy  is  a significant  improvement. 

7 . 3 Use  of  Fresh  Data 

We  recommend  that  data  also  be  collected  during  the  development  phase. 

This  could  be  done  in  larger  systems  by  automatic  collection  of  data  during 
compilation  and  testing  and  would  allow  important  feedback  to  the  developers 
that  would  allow  improvements  to  be  made  early  enough  to  have  an  effect  on 
the  software  reliability.  This  feedback  of  "fresh"  data  could  be  used  to  pro- 
vide improvements  in  the  areas  of  training  and  development  tools.  For  example, 
a high  incidence  of  improperly  formatted  data  errors  might  indicate  that  fur- 
ther training  in  the  data  definition  capability  of  the  HOL  in  use  is  necessary. 
In  the  subject  project,  Input/Output  software  had  a high  incidence  of  soft- 
ware errors  (36  SPRs/1000  Source  Lines).  This  can  partially  be  attributed  to 
the  fact  that  different  configurations  of  hardware  required  different  I/O 
coding.  It  is  also  probable  that  the  fact  that  the  Digital  Simulator  had  no 
I/O  simulation  capability,  caused  software  to  be  released  to  integration  testing 
without  actually  exercising  the  I/O  code.  This  feedback  early  in  the  project 
could  have  resulted  in  I/O  simulation  being  added  to  the  Digital  Simulator. 
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This  potential  feedback  benefit  would  also  justify  the  collection 
during  the  development  process  rather  than  "after-the-fact,"  and  therefore 
increase  its  own  reliability. 
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APPENDIX  A 

DATA  BASE  DESCRIPTION  FILE  FORMATS 


File  #1  Software  Module  Descriptions 


The  Software  Module  Description  file  contains  software  descriptive  data 
and  consists  of  one  record  per  module.  It  is  used  to  validate  file  // 2 data 
and  provide  statistics. 


Record  Format: 


Columns 

Field 

Code 

1 

File  Identification 

ll^fl 

2-6 

Project  Identification 

Alphanumeric 

7-8 

Project  Code 

Alphanumeric 

9-15 

Module  Identification 
(left  justified) 

Alphanumeric 

16-17 

Version  Identification 

Alphanumeric 

18 

Module  Function 

Alphanumeric 

X = Control 
P = Input/Output 
L = Logical 
D = Data  Manipulation 
M = Mathematical 
T = Test  Driver 
C = Confidence  Test 
B = Data  Base 
0 = C0MP00L 
R = Microcode 

19 

Module  Complexity 

Alphabetic 

S = Simple 
M = Medium 
C = Complex 

20 

Source  Language 

Alphabetic 

A = Assembler 
J = JOVIAL 
F = Fortran 
D = Direct  Code 

21-25 

//  Source  Statements 

Numeric 

Right  justified 


Columns 


Field 


Code 


Object  Size 

Including  literals  and 
local  data.  Not  including 
buffers.  Must  be  in  deci- 
mal. Right  justified. 

Mode  of  Construction 

0 = Unstructured 

1 = Modular 

2 = Top  Down 

3 = Modular  Top  Down 

4 = Structured 

5 = Modular  Structured 

6 = Top  Down  Structured 

7 = Modular  Top  Down 

Structured 


Numeric 


Numeric 


File  #2  Software  Problem  Reports 

This  file  consists  of  data  from  Software  Problem  Reports  and  Software 
Modification  Notices  and  consists  of  one  record  per  module. 


Record  Format: 


Columns 


Field 


13-19 


File  Identification 

Project  Identification 

Project  Code 

SPR  Number 

Right  justified 
Blank  if  no  SPR# 

Module  Affected 
Identification 

Left  justified 

Version  Identification 


Alphanumeric 

Alphanumeric 

Numeric 


Alphanumeric 


Alphanumeric 
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Record  Format: 


Columns 

Field 

Code 

22-29 

Date  SPR  Opened 

Alphanumeric 

(MM/DD/YY) 

Blank  if  no  SPR 

30 

Termination  Code 

Alphabetic 

Blank  = Terminated 
Normally 
S = Software 
Aborted 
H = Hardware 
Aborted 

31  Seriousness  of  Problem  Numeric 

1 = Critical 

2 = Medium  priority 

3 = Low  priority 

A = Suggested  important 

32  Test  Period  Alphabetic 

D = Development  - 
Unit  Test 

V = Validation  - 

Unit  Acceptance 

I = Integration 

A = Acceptance  of  Build 

0 = Operational 
Demonstration 


33-37 

Error  Category 

Alphanumeric 

38-41 

Applicable  SMN  Number 

Numeric 

42-46 

Type  of  Correction 

Alphabetic 

New  Module  Update 
X in  Col  42 

Document  Update 
X in  Col  43 

Record  Format: 


Columns 

42-46 


47-54 

55-57 


Field 


Code 


COMPOOL  Change 

X in  Col  44 

Data  Base  Change 

X in  Col  45 

Explanation 

X in  Col  46 

Leave  column  blank 
if  not  applicable.  Use 
more  than  one  type  if 
several  apply. 

Date  SPR  Closed  Alphanumeric 

(MM/DD/YY) 

The  SPR  is  closed 
by  an  SMN,  therefore, 
this  data  is  taken  from 
the  SMN. 

Days  Open  Numeric 

Total  of  working 
days  between  the  open 
and  closed  date.  If  only 
an  SMN  appears  it  reflects 
1 day  open. 

Right  justified. 
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File  If 3 Error  Categories 


This  file  contains  the  error  categories  and  descriptions.  It  is  used  to 
validate  file  #2  data  and  is  listed  for  reference.  It  consists  of  one  record 
per  error  category. 


Record  Format: 


Columns 

1 

2-6 

7-80 


Field 


File  Identification 
Error  Category 
Error  Description 


Code 


H3H 

Alphanumeric 

Alphanumeric 
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APPENDIX  B 

SOFTWARE  MODULE  DESCRIPTIONS 
FILE  NO.  1 LISTING 


B-l 


SOuBCF  LANGUAGE  30u»CE  SIZE  OBJFCT  SIZE  MODE  nF  CONSTRUCTION  n*S*RS 


000  UNSTRUCTURED 


OOOCOOOCOOODOOQCO  o O O O OO  C Ci  t 

UiUJUJiwuJaiiiiiuuJUjUiliiiiJIiMiJiuaj  ujujuj  uj  ujuj  u,  i 

“^“““““““““•““ttttattttaaaaaaaaaaaaaaaaai 

33  3 3 3 3 3 3 ^ M u < « 3 3 3 ^ U ■*  3 O « 3 


u u u u 
3 3 3 3 < 
m a a ( 
*-*-*-*-  j 
in  n i/i  vi 

3 3 3 3 


3 3 3 3 O « 
0 0 0 0 3: 

o o co  o a t 

* * a i t-  * 


^^-^^^^^^333_,33333_.0OQO0333030330330 
Il^lfflraiia  O O O o O a a arXArirxanorarn 


3^^3333333333333333 

roaaxaaxaaxaaaaaaaa 

-a 

> Wioioininnifliflinwini/nnininiflio 
’ ^77/777^/777777/7 

5 33333333333333333 


33333333_»333333000003330303303 

xaaxaaxaaaaaaaaoooooaaaraaotraca 

*_*“>“*_*_*-*-*-*-*-»-*-*-*-*-i  a a a a »-»-  »-  a t-  x »-  »-  a »- 

loinwwifliflinwiniflwinmmm  mm  m «o  «/>  <n  ®» 


m m m m 

7 7 7 7 

3 3 3 3 


IUUUUU 
13  3 3 3 3 

i a a a cr  a 

*-*-*-►-►- 
m m m mm 

7 7 7 7 7 

3 3 3 3 3 


^ ^ Jo  IT  1 ^ ^ ^ ^ m ^ — a — ®«~o7uaac’cm®<va  a «;  f*-  *v  m m a — • r~ 

^o.  «mo/)a;rur-r\joa  / a a ■>  a it  nj  o c m ^ ff 

~ ^ — a f-o  a — — ^ j ivj  -«  *a  — o ~ a®  ax— ocmx.A'*  „ 


« r\j  o 9 '*7>'*S044)40r'9 
®3h(\i'-omoni3ircinfff- 

— — IT 


*-  fi  f\i  o n «mmm»  ® <\i»  x Cfyioonjin-*®®  a — x m j f~  |\<  I r-  O « p#>  o 
’ ® 3'Vf\i-3mc»ni/'»'irfvef\(cmer-i\io»  3«3hr'ir--m« 

Of\l-*— • — OO— •— • — • r-~  jl  r\j  o ® O a o — • *^  7 m s - m 

- " •*  — m --•  o aj  <\i  « — .rv,— n>c.<v~-<v^-  r- 


>aj>>>>>aa  >>>: 

C!»-«OOOOOIaiUjCOOC 

•aOTVTVimim-)-!' 

in  m 


x i a ia  i u.  a a a a a ii  il  i iuu  u.  i a ^u-u. 

V*J«  333333333  33333—  3 _|  _J  3 

— * • *-  a *-a  *-*-*-*-*-a  a *-aa  q •—  •—  a.  a a 

cioioocooiici  j a r c i xa 

UJOluCUJibiiiliiUiG*'IU>'»-t-iuUjO>*>- 
aoaoaa  xa  a o in  a mm  m a a o m in 


m m mcrirmirmr*) 


3 m •-  »~ 

•AitAJ  ^ *.  iLOilAlliaildlLIlt  «*  U.V-W.V- 

UUUUUUUUUUUUUIUVUV 
77^-7  7i  7 / 77777  7 f 7 T 

~UU.UUUUu.UU  < U.  I®  u.  Ia 

tLCCfCtC'  »-  ( I C ! 

tiUUUUUUu  U U f U X u •* 

7 7 7 7 7 7 7.  / . / » 7 J 

CfCCCCCCCf  C t 

l-J  o u o u u u u u u u t> 


O I 

•“■  • am  £>  t,  c — m ^ it  cr-  -j  n o — <\j  •*>  am  o x a c-  — a.  m x m c t ■>  c x \ m x m « ►-  «.  a o -«  ai  »n  x 

• n -c  n-nnn.r^*'^*~'''~'-  r-  r-  a-  * rr  xni  an  a x a a a a a a > a a a coococooco----- 

ui  • ooorccoooocococ  ocooooooconoci  r>ocnoor.  — — ~ — — — ^ 

j j oouoeu  woeotfoiii;  L'ooueeoooe  l*  uc  i"  c lu*c  cir  irue  uoooea  e ouooooc 

J i o u p o r.  n D o c o o o d c n r.  o o c c r :j  r:  ci  ; : c 3 • c - r r r-  < ('c.  ronrooocooooocM 

t iffffott(»#ua(»ooort(»B(jffo»aao»onoaon5ftonoooftttHaa»n(»irttttftQ(i(» 
oiftoaft&Q(iQoaaannn«aftanaaanaiiQaaaaaannQ(V9aaaftooaaQaftftB 


xoouce  io  vfnsro*.  »noi/i.F  Fj^crrov  eo-cif  <m  sou»cr  i**su*Gt  so j»ce  sm  o*jfct  sm  -oof  of  cfl*JT»jcrro«,  \.s»«s 


r 


a if  c c c r 


• O O GOOD 

i luw  tot  t*i  M fci 

i a s • tt  a s a 

ft  3 3 ft  3 3 5 3 

I ► ► -JS-  ► 

ft  CJ  CJ  3 O U U U 
ft  3^03333 

• > ft  oa  a a t 

i <n  a « a « « 

» * r //// 

ft  3 3 3 3 3 3 

ft 


I ClO  7-C7  < 

i o « - « x -a 

i c IT  it  r- 

ft  IT*  I/'  ow 


— »r  ft  ft  » e 

»OH*l  -•  ft  3 
-*  fU  — 


> > > > a > > 
o o o o o o 


ft 

I I 2 I 7 I IL  7 

ft  3 =3  13  O 3 _»  35 

i a *- 

I coooox  o 

I lu  hi  lu  lu  ui  m lu 

I 7 I 2 I T SO  2 


ft  VO  V) 

I Ul  IU 
I ft-  ft-  l*J 

I •»  _»  _i  _l  _l 

I ILkl  4 K O 4 

I uuauoft  u 

I 7 2 

I lu  k.  4 u e 7 li 

• CCft-CCCC 
I ►«  *-  * _l  _i  o _» 

ft  u.  u.  o 

• z r 
ft  c c 

• u u 


I l/'  ft  F>  C ft  O — 
I — — — — — f\j  A, 


ft  O 15  li  O U (3  l? 
ft  OOOOOOO 

I ft  (t  ft  ft  O’  D D 
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APPENDIX  C 

SOFTWARE  PROBLEM  REPORTS 
SAMPLE  OF  FILE  NO.  2 LISTING 


c-i 


CO«»ECTI^ 

TvPf 


APPENDIX  D 

ERROR  CATEGORIES  (FAULT  TAXONOMY) 
FILE  NO.  3 LISTING 


D-l 


LLO  70  DAT*  9*SE  “ANAGE»ENT  AND  InTEGRTTt 

CLORO  EXTERNAL  PROGPA"  INTERFACE 

LLORO  “ODIFTCATION  FOR  SPECIAL  TEST  PURPOSES 


ISJjsr.  c»»nj  t*  CO«i»!LEO  PROGH»m 


APPENDIX  E 

STATIC  STATISTICS  FOR  JOVIAL  SOURCE  MODULES 


Nine  modules  were  examined  by  the  U1108  JOVIAL  program  (STATGT)  to  see 
how  frequently  certain  statements  are  used  in  practice.  Tables  E-l  and  E-2 
show  the  distribution  of  statement  types.  Also,  calculations  are  provided 
for  executable  statement  types.  Certain  changes  were  made  to  the  data  to 
eliminate  discontinuities*.  The  most  frequently  used  language  construct  is 
the  = sign.  This  is  because  of  its  use  in  the  assignment  statement  (23  per- 
cent) . The  next  most  used  construct  is  subscription  (14  percent)  , followed 
by  GOTO  (8  percent)  and  IF  (8  percent) . Nothing  can  be  said  about  the  pro- 
cedure call  mechanism  because  the  same  construct  is  used  for  other  features. 
The  BEGIN-END  delimiters  are  used  about  6 percent  of  the  time.  This  implies 
some  blocking  in  the  decision  making  logic.  The  EQ  relational  operator  was 
most  highly  used  (5  percent) . The  most  used  executable  statements  were 
assignment  (54  percent),  IF  (19.7  percent),  and  GOTO  (19.6  percent). 

A typical  program  consisted  of  assignment  statements  and  blocked  condi- 
tion checking  statements.  Programming  with  the  use  of  tables  appears  to  be 
prevalent.  Some  explicit  loops  are  seen.  Bit  and  byte  manipulation  do  not 
appear  to  be  frequently  used. 


*See  Note  3 of  Table  E-l. 


E-l 


TABLE  E-  1 


DISTRIBUTION  AND  MODULE  USAGE  OF  STATEMENT  TYPES 
(9  OPERATIONAL  MODULES) 


ASSIGN 

25 

0.  37 

BIT 

57 

0.  85 

BYTE 

97 

1.  4 

3 

438 

. - 

’ 3 

330 

- - 

s3 

3251 

- - 

BEGIN-END 

401 

5.  98 

START-TERM 

9 

0.  13 

DIRECT- JOVIAL 

71 

1.  06 

($-$) 

929 

13.  8 

ITEM 

438 

6.  5 

TABLE 

26 

0.  38 

ARRAY 

4 

0.  06 

PROC 

20 

0.  3 

SWITCH 

14 

0.  2 

OVERLAY 

6 

0.  09 

’ PROGRAM 

0 

0 

BLOCK 

0 

0 

Subtotal 

10733 

le  ss 

4019 

Total 

6714  100 

1)  expression  grouping,  procedure,  function  c 

all 

2)  assignment,  FOR,  procedure  call  parameter  delimiting 

3)  deleted  from  total  for  reasons  of  ambiguity 

TABLE  E-2 

DISTRIBUTION  AND  MODULE  USAGE  OF  EXECUTABLE  STATEMENTS 


E- 


APPENDIX  F 

CONSTITUENT  PROGRAM  MODULES 
OF  BUILDS  "F"  AND  "G" 


Refer  to  Appendix  B (Software  Module  Descriptions)  for  further  informa- 
tion about  each  of  these  modules  listed. 


Build  F - Operation  Build  (55  modules) 


PROG001, 

6,  8,  9 

, 11 

, 12 

, 13 

, 14 

, 15,  16,  20 

, 21 

, 24 

27, 

28,  29, 

36, 

39, 

41, 

43, 

45,  46,  50, 

52, 

53, 

58, 

59,  60, 

62, 

64, 

65, 

66, 

67,  72,  75, 

76, 

81, 

<r 

00 

86,  87, 

88, 

92, 

95, 

106 

, 108,  110, 

111, 

112 

114,  117,  118,  119. 

Build  G - Initialization  Build  (25  modules) 


PROG002 , 57,  70,  71,  77,  79,  85,  89,  91,  93,  94, 

96,  97,  98,  99,  100,  101,  102,  103,  104,  105, 
107,  109,  116,  120. 
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METRIC  SYSTEM 


BASE  UNITS: 

Quantity 

length 

mass 

time 

electric  current 
thermodynamic  temperature 
amount  of  substance 
luminous  intensity 

SUPPLEMENTARY  UNITS: 

plane  angle 
solid  angle 

DERIVED  UNITS:  * 
Acceleration 

activity  (of  a radioactive  source) 

angular  acceleration 

angular  velocity 

area 

density 

electric  capacitance 

electrical  conductance 

electric,  field  strength 

elec  tric  inductance 

electric  potential  difference 

electric:  resistance 

electromotive  force 

energy 

entropy 

force 

frequency 

illuminance 

luminance 

luminous  flux 

magnetic  field  strength 

magnetic  flux 

magnetic  flux  density 

magnetomotive  force 

power 

pressure 

quantity  of  electricity 
quantity  of  heat 
radiant  intensity 
specific:  heat 
stress 

thermal  conduc  tivity 
velocity 

viscosity,  dynamic 
viscosity,  kinematic 
voltage 


SI  Symbol 


Formula 


metre 

kilogram 

second 

ampere 

kelvin 

mole 

candela 


radian 

steradian 


metre  per  second  squared 

disintegration  per  second 

radian  per  second  squared 

radian  per  second 

square  metre 

kilogram  per  cubic,  metre 

farad 

siemens 

volt  per  metre 

henry 

volt 

ohm 

volt 

joule 

joule  per  kelvin 

newton 

hertz 

lux 

candela  per  square  metre 
lumen 

ampere  per  metre 

weber 

tesla 

ampere 

watt 

pascal 

coulomb 

|oule 

watt  per  steradian 

joule  per  kilogram-kelvin 

pascal 

watt  per  metre-kelvin 

metre  per  second 

pascal-second 

square^  metre  per  second 

volt 


volume 

cubic  metre 

wavenumber 

reciprocal  metre 

work 

joule 

1 

SI  PREFIXES: 

Multiplic  ation  ka<  tors 

Prefix 

1 000  000  000  000 

10  ii 

tc*ra 

1 000  000  000 

K 1 Ka 

1 000  000 

mega 

1 000 

- ] o' 

kilo 

100 

rr  Jf)l 

hecto* 

10 

= 10* 

deka* 

0 1 

= 10  1 

deci* 

0 01 

1 o i 

centi* 

0 001 

= 10  1 

milli 

0 I 00  001 

= 10“  * 

micro 

0 000  000  001 

=S  10“  * 

nano 

0.000  000  000  001 

— 1 0 * * 

pic  o 

0 000  000  000  000  001 

HI- 

femto 

0 000  000  000  000  000  001 

- 1 (1  ~ •• 

HttO 

m/s 

(disintegration)^ 

rad/s 

rads 

m 

kg/m 

A-s/V 

A/V 

\i  m 

V-s/A 

W'A 

V/A 

W/A 

N-m 

I'K 

kg-m/s 

(cycle)/s 

lm/m 

cd/m 

cd-sr 

A;m 

V-s 

Wb/m 

|/s 

N/m 

A-s 

N-m 

Wsr 

I'kg.K 

N'm 

W'nvK 

ms 

Pa-s 

ms 

W'A 

m 

(wave)/m 

N-m 


SI  Symbol 


To  be  avoided  where  possible 


